Skip to content

docs: move experiments from Internals to Contributing section#2853

Merged
waynesun09 merged 4 commits into
mainfrom
fix-2757-experiments-docs
Jul 2, 2026
Merged

docs: move experiments from Internals to Contributing section#2853
waynesun09 merged 4 commits into
mainfrom
fix-2757-experiments-docs

Conversation

@waynesun09

@waynesun09 waynesun09 commented Jul 1, 2026

Copy link
Copy Markdown
Member

Summary

  • Add symlink docs/experiments → ../experiments so VitePress discovers experiment markdown from the submodule
  • Move "Experiments (Exploratory)" nav entry from Internals to Contributing — experiments are community/contributor content, not internal implementation details
  • Add submodules: true to site-build.yml checkout step so the experiments submodule is available during CI builds
  • Remove misplaced docs/superpowers/experiments/ directory (was under Internals/superpowers, which is wrong)
  • Add predev/prebuild hooks to website/package.json that auto-run git submodule update --init, so local npm run dev and npm run build match CI behavior without manual steps
  • Rename docs/site-deployment.mddocs/web-admin-deployment.md, mark web admin SPA as on hold, remove doc-site content
  • Add docs/doc-site.md — lightweight VitePress doc site guide (local dev, build, CI, submodules)
  • Update sidebar nav and all cross-references

Test plan

  • VitePress build succeeds with 104 experiment pages under /experiments/
  • Experiments appear in the Contributing nav section, not Internals
  • Old /superpowers/experiments/ path no longer exists in build output
  • CI site-build workflow passes with submodule checkout
  • npm run prebuild correctly initializes submodules from website/ directory
  • doc-site.html and web-admin-deployment.html in build output, no site-deployment.html
  • Sidebar shows "Doc Site" and "Web Admin Deployment (On Hold)" entries

Fixes #2757

@waynesun09 waynesun09 requested a review from a team as a code owner July 1, 2026 16:03
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

docs: move experiments from Internals to Contributing section

📝 Documentation ⚙️ Configuration changes 🕐 Less than 10 minutes

Grey Divider

AI Description

• Add docs/experiments symlink pointing to ../experiments submodule so VitePress discovers
 experiment markdown files during builds
• Move "Experiments (Exploratory)" nav entry from the Internals section to the Contributing section
 in VitePress config, reflecting that experiments are community/contributor content
• Add submodules: true to the site-build.yml checkout step so the experiments submodule is
 available in CI
• Remove the misplaced docs/superpowers/experiments/oauth-localhost-part-b/ directory that was
 incorrectly nested under Internals/superpowers
Diagram

graph TD
    A["site-build.yml"] -->|"checkout submodules: true"| B["experiments submodule"]
    B -->|"symlink docs/experiments"| C["VitePress docs/"]
    C --> D["config.ts nav"]
    D -->|"removed from"| E["Internals section"]
    D -->|"added to"| F["Contributing section"]
    G["docs/superpowers/experiments/"] -->|"deleted"| H["(removed)"]

    subgraph Legend
      direction LR
      _ci["CI Workflow"] ~~~ _sub(["Submodule"]) ~~~ _del{{"Deleted"}}
    end
Loading
High-Level Assessment

The PR's approach is optimal: using a symlink to expose the submodule to VitePress is the standard pattern for this kind of setup, and the nav reorganization correctly reflects the contributor-facing nature of experiments. No meaningful alternatives exist.

Files changed (3) +7 / -5

Documentation (1) +5 / -5
config.tsMove Experiments nav entry from Internals to Contributing section +5/-5

Move Experiments nav entry from Internals to Contributing section

• Removes the 'Experiments' nav item from the Internals sidebar (which pointed to 'superpowers/experiments') and adds an 'Experiments (Exploratory)' nav item to the Contributing sidebar pointing to the new 'experiments' path.

website/.vitepress/config.ts

Other (2) +2 / -0
site-build.ymlAdd submodules: true to checkout step for CI builds +1/-0

Add submodules: true to checkout step for CI builds

• Adds 'submodules: true' to the 'actions/checkout' step so the experiments git submodule is checked out during CI site builds, making experiment markdown available to VitePress.

.github/workflows/site-build.yml

experimentsAdd symlink docs/experiments → ../experiments submodule +1/-0

Add symlink docs/experiments → ../experiments submodule

• Creates a symlink from 'docs/experiments' to the '../experiments' submodule directory, allowing VitePress to discover and render experiment markdown pages under the '/experiments/' path.

docs/experiments

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown

Site preview

Preview: https://f2576b35-site.fullsend-ai.workers.dev

Commit: 48683640c22a6a82a570b77cac36aadb84560562

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 1, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 4:06 PM UTC · Completed 4:17 PM UTC
Commit: 8551b84 · View workflow run →

@qodo-code-review

qodo-code-review Bot commented Jul 1, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📎 Requirement gaps (1) 📜 Skill insights (0)

Context used
✅ Compliance rules (platform): 54 rules

Grey Divider


Action required

1. Experiments not under docs/contributing/ 📎 Requirement gap ≡ Correctness
Description
The docs nav and discovery logic points experiments to docs/experiments with URL base
/experiments, rather than sourcing them from docs/contributing/experiments as required. This can
cause experiments to appear under the wrong generated path and violates the required docs-tree
location for experiments content.
Code

website/.vitepress/config.ts[R250-254]

+            {
+              text: 'Experiments (Exploratory)',
+              collapsed: true,
+              items: getMarkdownFiles('experiments', 'experiments'),
+            },
Relevance

⭐⭐ Medium

No clear prior reviews on required experiments location/pathing; closest evidence only shows general
VitePress nav/link tweaks accepted (#2765).

PR-#2765

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The checklist requires experiments to be provided via docs/contributing/experiments/ and appear
under Contributing from that path. The current VitePress config reads from docs/experiments and
generates links under /experiments, which does not meet the required
docs/contributing/experiments location/path.

Experiments docs appear under docs/contributing/experiments in the generated docs nav
Docs build inputs include experiments markdown via docs/ tree (copy/mount/symlink)
website/.vitepress/config.ts[7-30]
website/.vitepress/config.ts[250-254]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Experiments are currently sourced from `docs/experiments` and linked under `/experiments`, but compliance requires the docs-visible location to be `docs/contributing/experiments/` and for navigation to be sourced from that path.

## Issue Context
`getMarkdownFiles(dir, base)` reads from `docs/<dir>` and generates links under `/<base>/...`, so using `('experiments','experiments')` will not satisfy the required `docs/contributing/experiments` structure.

## Fix Focus Areas
- website/.vitepress/config.ts[7-30]
- website/.vitepress/config.ts[250-254]
- docs/contributing/experiments[1-1]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. No directory check before readdir 🐞 Bug ☼ Reliability
Description
The new sidebar entry calls getMarkdownFiles('experiments', ...), but getMarkdownFiles only checks
existsSync() and then calls readdirSync(); if the path exists but is not a directory, the docs build
will throw ENOTDIR and fail. This becomes more likely with a symlink-based layout (unsupported
symlinks often checkout as regular files).
Code

website/.vitepress/config.ts[R250-254]

+            {
+              text: 'Experiments (Exploratory)',
+              collapsed: true,
+              items: getMarkdownFiles('experiments', 'experiments'),
+            },
Relevance

⭐⭐ Medium

No historical evidence found for adding isDirectory/stat checks before readdirSync in VitePress
config.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
getMarkdownFiles() currently checks existsSync(fullDir) and then immediately calls
readdirSync(fullDir), with no verification that fullDir is a directory. The PR adds a new
sidebar section that invokes getMarkdownFiles('experiments', 'experiments'), increasing the chance
that a non-directory docs/experiments path crashes the docs build.

website/.vitepress/config.ts[6-31]
website/.vitepress/config.ts[224-255]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`getMarkdownFiles()` assumes `fullDir` is a directory after `existsSync(fullDir)`. If `fullDir` exists as a non-directory (file, broken symlink materialized as file, etc.), `fs.readdirSync(fullDir)` will throw and break the site build.

## Issue Context
This PR adds a new sidebar section that calls `getMarkdownFiles('experiments', 'experiments')`, so any path/type mismatch under `docs/experiments` can now fail the docs build.

## Fix Focus Areas
- website/.vitepress/config.ts[9-31]
- website/.vitepress/config.ts[247-256]

## Suggested fix
- After resolving `fullDir`, add a `try { const st = fs.statSync(fullDir); if (!st.isDirectory()) return [] } catch { return [] }` guard before calling `readdirSync`.
- Optionally also wrap `fs.statSync(entryPath)` in a try/catch to avoid a single bad entry (e.g., broken link) crashing the whole sidebar generation.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

3. Stale experiment doc links ✓ Resolved 🐞 Bug ≡ Correctness
Description
This PR deletes the oauth-localhost experiment helper under docs/superpowers/experiments, but
existing docs still link to that removed path, producing broken links in the rendered site. Because
the VitePress config sets ignoreDeadLinks=true, CI will not fail, so the regression is likely to
ship unnoticed.
Code

docs/superpowers/experiments/oauth-localhost-part-b/serve.py[L1-20]

-#!/usr/bin/env python3
-"""Local OAuth Task 1 helper: inject CLIENT_ID and exercise the GitHub token exchange.
-
-Requires CLIENT_ID in the environment (GitHub App OAuth client id — public).
-
-  export CLIENT_ID='Iv23…'
-  python3 serve.py
-
-Optional CLIENT_SECRET (same machine only — never commit, never send from the browser):
-when set, POST /_experiment/github-access-token performs the full documented web-app
-exchange (includes client_secret server-side only).
-
-Optional: PORT (default 5173), BIND (default 127.0.0.1). If PORT is not 5173, register the
-matching callback URL on your GitHub App (http://localhost:<PORT>/oauth/callback.html).
-"""
-
-from __future__ import annotations
-
-import json
-import os
Relevance

⭐⭐⭐ High

Team repeatedly fixes broken/stale doc links (accepted in PR #2009, #601; VitePress link fixes in
#2765).

PR-#2765
PR-#2009
PR-#601

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Two docs files still link to ../experiments/oauth-localhost-part-b/... relative to
docs/superpowers/{plans,specs}, which resolves to the deleted docs/superpowers/experiments/...
path. The VitePress config explicitly disables dead-link checking, so the broken links will not fail
CI.

docs/superpowers/plans/2026-04-12-fullsend-admin-spa.md[105-112]
docs/superpowers/specs/2026-04-06-fullsend-admin-spa-design.md[173-180]
website/.vitepress/config.ts[135-142]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Docs pages still link to `../experiments/oauth-localhost-part-b/...` from within `docs/superpowers/*`, but this PR removes `docs/superpowers/experiments/oauth-localhost-part-b/`. Those relative links now resolve to a non-existent location and will 404 in the built docs.

## Issue Context
The docs build is configured with `ignoreDeadLinks: true`, so this won’t be caught during CI.

## Fix Focus Areas
- docs/superpowers/plans/2026-04-12-fullsend-admin-spa.md[105-112]
- docs/superpowers/specs/2026-04-06-fullsend-admin-spa-design.md[173-180]
- website/.vitepress/config.ts[135-142]

## Suggested fix
Update the markdown links to point at the new experiments location that will exist after this PR (e.g. `../../experiments/oauth-localhost-part-b/serve.py` / `../../experiments/oauth-localhost-part-b/README.md` if experiments are surfaced via `docs/experiments`, or replace with a stable GitHub URL to the experiments repo if that’s the intended source of truth).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Qodo Logo

Comment thread website/.vitepress/config.ts
Comment thread website/.vitepress/config.ts
Comment thread docs/superpowers/experiments/oauth-localhost-part-b/serve.py
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 1, 2026

Copy link
Copy Markdown

Review

Findings

Medium

  • [submodule-trust] .github/workflows/site-build.yml:22 — Adding submodules: true causes ALL submodules in .gitmodules to be cloned, including eval/.agent-eval-harness from https://github.com/ralphbean/agent-eval-harness.git (branch worktree-execution-hooks) — a third-party repository outside the fullsend-ai org. Although the build steps do not reference that submodule, its contents are present on the runner's filesystem during the entire workflow. The third-party repo targets a non-default branch, which receives less community scrutiny.
    Remediation: Replace submodules: true with manual selective checkout: after the checkout step, add run: git submodule update --init experiments.

  • [stale-path-reference] docs/superpowers/plans/2026-04-12-fullsend-admin-spa.md:109 — The link to oauth-localhost-part-b/serve.py was updated to a commit-pinned absolute GitHub URL (which will resolve), but the surrounding prose still says "the repo includes" the file — this is misleading after deletion of the oauth-localhost-part-b/ directory.
    Remediation: Update the prose to indicate the experiment is archived, e.g., "The repo previously included" or "See the archived experiment at".

  • [protected-path] .github/workflows/site-build.yml — This file is under .github/, a protected path requiring human approval. The PR links to issue Docs site: experiments/ submodule content not included in built site #2757 and explains the rationale (adding submodules: true for experiments submodule checkout during CI site builds). Human approval is always required for protected-path changes regardless of context.

Previous run

Review

Findings

High

  • [logic-error] .gitmodules — The experiments submodule uses an SSH URL (git@github.com:fullsend-ai/experiments.git). actions/checkout with submodules: true uses HTTPS token auth by default. Unless the workflow is configured with an SSH key or the submodule URL is overridden to HTTPS, the submodule checkout will fail with an authentication error, breaking the site-build workflow on every PR and push.
    Remediation: Change the submodule URL in .gitmodules from SSH to HTTPS (https://github.com/fullsend-ai/experiments.git), or configure the workflow with an SSH deploy key, or add insteadOf git config in the workflow.

  • [stale-path-reference] docs/superpowers/plans/2026-04-12-fullsend-admin-spa.md:109 — References deleted experiment path ../experiments/oauth-localhost-part-b/serve.py. After this PR, the oauth-localhost-part-b directory under docs/superpowers/experiments/ is deleted, so this link will be broken.
    Remediation: Update the reference to reflect the new experiments location or remove the reference.

  • [stale-path-reference] docs/superpowers/specs/2026-04-06-fullsend-admin-spa-design.md:177-178 — References deleted experiment with relative links to ../experiments/oauth-localhost-part-b/serve.py (line 177) and ../experiments/oauth-localhost-part-b/README.md (line 178). The oauth-localhost-part-b directory is being removed in this PR.
    Remediation: Remove or update references to oauth-localhost-part-b.

Medium

  • [submodule-trust] .github/workflows/site-build.yml:22submodules: true clones ALL submodules defined in .gitmodules, including eval/.agent-eval-harness from github.com/ralphbean/agent-eval-harness.git — a third-party repository outside the fullsend-ai org. While no build steps currently execute code from that submodule, it expands the trust surface.
    Remediation: Consider selective submodule checkout via manual git submodule update --init experiments instead of submodules: true.

  • [protected-path] .github/workflows/site-build.yml — This file is under .github/ which is a protected path requiring human approval. The PR links to issue Docs site: experiments/ submodule content not included in built site #2757 and explains the rationale (adding submodules: true for experiments submodule checkout during site builds). Human approval is always required for protected-path changes regardless of context.


Labels: PR modifies docs site navigation config and adds submodule checkout to the CI site-build workflow.

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added component/docs User-facing documentation component/ci CI pipelines and checks submodules Pull requests that update submodules code labels Jul 1, 2026
@waynesun09 waynesun09 force-pushed the fix-2757-experiments-docs branch from 654e3cd to b71ccd3 Compare July 1, 2026 16:56
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 1, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 5:00 PM UTC · Ended 5:05 PM UTC
Commit: f417572 · View workflow run →

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 1, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 5:08 PM UTC · Ended 5:19 PM UTC
Commit: f417572 · View workflow run →

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 1, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 5:22 PM UTC · Ended 5:31 PM UTC
Commit: f417572 · View workflow run →

The experiments/ submodule content was not appearing on the docs site
because the only experiment doc was misplaced at
docs/superpowers/experiments/ (Internals section), the doc builder
only scans docs/, and site-build.yml did not checkout submodules.

- Add symlink docs/experiments -> ../experiments so VitePress
  discovers experiment markdown from the submodule
- Move "Experiments (Exploratory)" nav entry from Internals to
  Contributing to reflect that experiments are community/contributor
  content, not internal implementation details
- Add submodules: true to site-build.yml checkout step
- Remove misplaced docs/superpowers/experiments/ directory

Fixes: #2757

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
Add predev and prebuild hooks to website/package.json so that
npm run dev and npm run build automatically run
git submodule update --init. This closes the gap between CI
(which uses actions/checkout submodules: true) and local builds
where submodule content was silently missing.

Document the submodule requirement in docs/site-deployment.md
for workflows that bypass npm scripts.

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
Rename docs/site-deployment.md to docs/web-admin-deployment.md and
mark web admin SPA work as on hold. Remove doc-site (VitePress)
content from the web admin deployment guide since they are separate
stacks (web/ + cloudflare_site/ vs website/).

Add docs/doc-site.md covering the VitePress documentation site:
local dev, build, CI workflows, and submodule handling.

Update sidebar nav and all cross-references in admin-oauth-worker,
cloudflare-pages plan, and design spec.

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
Group conceptual docs (ADRs, Design Documents, Experiments) together
before operational docs (Doc Site, Web Admin). Shorten "Web Admin
Deployment (On Hold)" to "Web Admin (On Hold)" for cleaner navigation.

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 1, 2026

Copy link
Copy Markdown

🤖 Review · ❌ Terminated · Started 5:34 PM UTC · Ended 5:46 PM UTC
Commit: f417572 · View workflow run →

@codecov

codecov Bot commented Jul 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@fullsend-ai-review fullsend-ai-review Bot dismissed their stale review July 1, 2026 17:46

Superseded by updated review

@fullsend-ai-review fullsend-ai-review Bot added the requires-manual-review Review requires human judgment label Jul 1, 2026
@fullsend-ai-review

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 5:34 PM UTC · Completed 5:46 PM UTC
Commit: 4868364 · View workflow run →

@waynesun09 waynesun09 requested review from ascerra and ralphbean July 1, 2026 17:51
@waynesun09 waynesun09 added this pull request to the merge queue Jul 2, 2026
Merged via the queue into main with commit 216b559 Jul 2, 2026
21 checks passed
@waynesun09 waynesun09 deleted the fix-2757-experiments-docs branch July 2, 2026 12:07
@fullsend-ai-retro

fullsend-ai-retro Bot commented Jul 2, 2026

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 12:10 PM UTC · Completed 12:22 PM UTC
Commit: 4868364 · View workflow run →

@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #2853 — docs: move experiments from Internals to Contributing section

Timeline: Human-authored PR by waynesun09 (Jul 1 16:03 → merged Jul 2 12:07). 4 commits over ~1.5 hours of iteration. Linked to issue #2757.

What happened

  1. 16:03 — PR opened. Review dispatched.
  2. 16:17fullsend-ai-review[bot] posted REQUEST_CHANGES with 3 medium findings: [submodule-trust] (CI clones all submodules including a third-party repo), [stale-path-reference], [protected-path]. Previous run had also found 2 high findings (SSH vs HTTPS submodule URL, stale paths) that the author had already fixed.
  3. 16:55–17:18 — Author pushed 3 more commits (auto-init hooks, doc split, sidebar reorder). Each push triggered a new review dispatch.
  4. 16:56–17:314 review runs cancelled in succession as each push superseded the previous in-flight review.
  5. 17:31–17:46 — Final review run completed successfully, but posted only a status comment ("Finished Review · Success") — no review body or findings.
  6. Jul 2 06:07rh-hemartin approved (no comment body). The only visible agent review at this point was the DISMISSED REQUEST_CHANGES from 16:17.
  7. Jul 2 12:07 — PR merged. The [submodule-trust] finding (use selective checkout instead of submodules: true) was never addressed.

Key observations

Proposals

One proposal below. Skipped proposals for debouncing (covered by #1014, #1418, #1422, #1452, #1331) and unresolved-finding alerting (covered by #2099, #2677, #1956).

Proposals filed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/ci CI pipelines and checks component/docs User-facing documentation requires-manual-review Review requires human judgment submodules Pull requests that update submodules code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docs site: experiments/ submodule content not included in built site

2 participants